Building a Production Evaluation Strategy for My RAG System

A RAG system is easy to demo but hard to trust.

The first version of my personal AI Q&A system worked well enough to feel useful. It could retrieve context from my portfolio, project notes, technical writing, and GitHub-related sources, then answer questions through a conversational interface. But once the system started becoming part of my portfolio, “it seems to work” was no longer a good enough standard.

The real question became: how do I know it keeps working as I change the prompt, update the retriever, ingest new content, switch models, or improve the chat experience?

That is where evaluation became necessary.

Why I Needed Evaluation

My RAG system is not a generic chatbot. It answers questions about me: what I build, how I think about product engineering, what projects I have worked on, and how I use AI in my workflow.

That makes correctness more personal and more product-specific.

A bad answer is not just technically wrong. It can misrepresent my experience, exaggerate my background, miss important projects, or give a vague answer when the system actually has enough context to be specific.

So I needed an evaluation strategy that checks more than “did the model produce text?”

I wanted to measure:

• Whether the answer is grounded in retrieved context
• Whether the retrieved chunks are actually relevant
• Whether important source material is being found
• Whether the assistant avoids unsupported claims
• Whether prompt or model changes cause regressions
• Whether different LLMs behave better for this specific product

The Evaluation Architecture

I separated the system into four related parts: tracing, dataset, execution, and scoring.

This separation matters.

LangSmith tracing gives visibility into individual RAG runs. The local runner executes the real system. RAGAS scores RAG-specific quality. LangSmith experiments compare system versions over time.

Keeping these responsibilities separate makes the evaluation system easier to extend. If I change the scoring method, I do not need to rewrite the runner. If I change the model or retriever, I can still use the same dataset.

Adding LangSmith Tracing

Before building experiments, I first added LangSmith tracing.

Tracing is useful because RAG failures are often hidden inside the pipeline. A weak answer might come from a bad query rewrite, poor retrieval, noisy reranking, missing context, or the final LLM ignoring the context.

Without tracing, I only see the final answer. With tracing, I can inspect the intermediate steps.

For this system, tracing helps answer questions like:

• What did the query rewrite produce?
• Which chunks were retrieved?
• Did reranking improve or hurt the result?
• What context was passed to the LLM?
• How long did each step take?
• Which model call produced the final answer?

That makes LangSmith useful even outside formal eval runs. It becomes a debugging layer for normal development.

If a user asks “What projects have you built?” and the answer is weak, I can inspect the trace and see whether the problem is retrieval, context construction, or generation.

This is especially important for a personal AI assistant because the failures are often subtle. The answer may sound fluent but still miss the most relevant project or overstate something that the context does not support.

Building the Dataset

The dataset is the foundation of the evaluation system.

I created a JSONL dataset with questions that represent the actual behavior I care about. Since this assistant is about me, the dataset needs to reflect the types of questions a visitor, recruiter, collaborator, or technical reader might ask.

The cases are grouped by tags:

• profile: questions about who I am and what I do
• skills: frontend, backend, AI, product, and full-stack engineering
• projects: questions about specific projects I have built
• comparison: positioning questions, such as product engineer vs. frontend engineer
• negative: questions the assistant should not answer without evidence
• multi_turn: follow-up questions that depend on conversation history

Each case includes the question, reference answer, expected sources, whether the system should answer, and tags.

This turns the dataset into a regression suite. When the system fails in a real interaction, I can add that question as a new eval case. Over time, the dataset becomes a record of the product’s expected behavior.

Local Evaluation

The local eval runner gives me a fast, repeatable way to test the system.

It runs the real RAG pipeline and writes structured output files. Those files include the generated answer, retrieved sources, and retrieved contexts. This is important because RAG evaluation is not only about the final answer. I also need to inspect what the system retrieved before answering.

The local runner calculates deterministic checks:

• answer_presence: answerable questions should produce an answer
• source_match: expected sources should appear in retrieved results
• abstention_correctness: unanswerable questions should avoid unsupported claims
• final_score: a simple aggregate for regression checks

These metrics are not meant to replace deeper evaluation. They are cheap guardrails. They catch obvious failures quickly before running more expensive LLM-based evaluation.

RAGAS Evaluation

RAGAS adds the RAG-specific metrics that deterministic checks cannot fully judge.

The core metrics I care about are:

• faithfulness: whether the answer is supported by the retrieved context
• answer_relevancy: whether the answer addresses the user’s question
• context_precision: whether the retrieved chunks are useful instead of noisy
• context_recall: whether the retrieved context covers the reference answer

These metrics help separate retrieval problems from generation problems.

If context_recall is low, the retriever may not be finding enough relevant information. If context_precision is low, the retriever may be pulling too much noise. If faithfulness is low, the model may be adding unsupported claims even when context is available.

That distinction is the main value of RAG evaluation. A single “answer quality” score is not enough. I need to know which part of the system is failing.

LangSmith Datasets and Experiments

After tracing and local evaluation, I added LangSmith dataset and experiment support.

The local JSONL dataset can be synced into LangSmith. Then I can run experiments against the same set of examples and compare versions of the system.

This is useful when comparing:

• prompt versions
• LLM providers
• answer models
• retrieval settings
• reranking behavior
• ingestion changes
• context formatting changes

Instead of manually asking a few questions after each change, I can run the dataset and compare results across versions.

This is especially useful for a RAG system because improvements are often tradeoffs. A prompt change might improve tone but reduce faithfulness. A retrieval change might improve project questions but hurt profile questions. Experiments make those tradeoffs visible.

Quality Gates

I treat evaluation scores as guardrails, not absolute truth.

The current minimum gates are:

• local final_score >= 0.80
• faithfulness >= 0.85
• answer_relevancy >= 0.80
• context_precision >= 0.75
• context_recall >= 0.70
• no silent regressions in negative or `project cases

The exact numbers can change as the dataset matures. What matters more is having a consistent baseline. If a new prompt, model, or retriever change lowers the scores, I need to understand why before accepting it.

The Development Loop

With tracing and evaluation in place, improving the RAG system becomes a tighter loop.

Before adding evaluation, improving the RAG system was mostly manual. I would ask a few questions, inspect the answers, and decide whether the result felt better.

That approach does not scale.

Now the system has a clearer feedback loop. Local checks catch obvious regressions. RAGAS evaluates groundedness and retrieval quality. LangSmith tracing explains what happened inside a run. LangSmith experiments help compare versions.

This moves the project from “a RAG demo that works” toward “a RAG product that can be maintained.”

Next Improvements

There are still several upgrades I want to add.

First, I want better tag-based reporting. Overall averages can hide weak areas. The system might score well overall while still performing poorly on project-specific questions or negative cases.

Second, I want a retrieval-only evaluation mode. RAGAS gives context precision and recall, but I also want direct checks for top-k source hits, query rewrite quality, and reranker impact.

Third, I want model comparison built into the runner. The system should make it easy to compare Google, OpenAI, DeepSeek, or other providers against the same dataset.

Fourth, I want better business-specific evaluators. For this project, general RAG metrics are not enough. I also care about whether the assistant represents me accurately, avoids overclaiming, and sounds like a useful portfolio assistant instead of a generic chatbot.

Fifth, I want stronger latency and cost tracking. A production RAG system should not only be correct. It should also be fast enough, reliable enough, and affordable enough to run continuously.

Finally, I want evaluation results to become part of the release process. If a future change lowers faithfulness, breaks project-related questions, or makes negative cases worse, that should be visible before the change ships.

Evaluation is not a one-time feature. It is part of the product loop.

The more the assistant becomes a real interface into my work, the more important it becomes to measure whether it is answering well, retrieving the right context, and improving without quietly regressing.